MULTI -ACCESS VIRTUAL PRIVATE NETWORK 




BACKGROUND OF THE INVENTION 



1 . Field of the Invention 

This invention relates a system and method for 
allowing private communications over an open network, and 
in particular to a virtual private network which provides 
data encryption and mutual authentication services for both 
client/server and peer-to-peer applications at the 
applications, transport driver, and network driver levels. 

2 . Discussion of Related Art 

A virtual private network (VPN) is a system for 
securing communications between computers over an open 
network such as the Internet. By securing communications 
between the computers, the computers are linked together as 
if they were on a private local area network (LAN) , 
effectively extending the reach of the network to remote 
sites without the infrastructure costs of constructing a 
private network. As a result, physically separate LANs 




can work together as if they were a single LAN, remote 
computers can be temporarily connected to the LAN for 
communications with mobile workers or telecommuting , and 
electronic commerce can be carried out without the risks 
5 inherent in using an open network. 

In general, there are two approaches to virtual 
private networking, illustrated in Figs. 1A and IB. The 
first is to use a dedicated server 1, which may also 
function as a gateway to a secured network 2, to provide 

10 encryption and authentication services for establishment of 
secured links 3 between the server 1 and multiple clients 
4-6 over the open network 7, represented in Fig. 1A as a 
cloud, while the second is to permit private communications 
links 8 to be established between any two computers or 

15 computer systems 9-12 on network 7, as illustrated in Fig. 
IB. 

The advantages of a client/server arrangement such as 
the one shown in Fig. 1A are that the server can handle 
functions requiring the majority of the computing 

20 resources, increasing the number of potential clients , and 
that management of the network, including key management is 
centralized. The disadvantage of a client/ server network 
of this type is that peer-to-peer communications links 
between applications on the client computers cannot utilize 

25 the security and management functions provided by the 
server, leaving such communications unprotected. On the 



10 



is 

=.y 



20 



other hand, the advantage of the direct peer-to-peer 
approach illustrated in Fig* IB is that it permits secured 
links to be established between any computers capable of 
carrying out the required security functions, with the 
disadvantages being the cost of configuring each computer 
to carry-out encryption, authentication, and key management 
functions, and the lack of central control. 

In both the client/ server and peer-to-peer approaches, 
a virtual private network can in theory be based either on 
applications level technology or can operate at a lower 
level. Generally, however, , peer-to-peer "tunneling" 
arrangements require modification of the lower layers of a 
computer's communications architecture, while client/server 
arrangements can use the applications level approach 
because less modification of the clients is required, and 
thus the two approaches are in practice mutually exclusive. 
The present invention, on the other hand, seeks to provide 
a virtual private network which utilizes a client/server 
approach, including centralized control of encryption, 
authentication, and key management functions, while at the 
same time enabling secured peer-to-peer communications 
between applications, by utilizing the server to provide 
authentication and session key generation functions for 
both client to server communications and peer-to-peer 
communications, providing a virtual private network capable 
of serving both as an extended intranet or wide area 
network (WAN), and as a commercial mass marketing network, 
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with high level mutual authentication and encryption 
provided for all communications. 

In order to completely integrate the two approaches 
and maximize the advantage of each approach , the invention 
5 maintains the applications level infrastructure of prior 
client server private networking arrangements , while adding 
shims to lower levels in order to accommodate a variety of 
peer-to-peer communications applications while utilizing 
the applications level infrastructure for authentication 
10 and session key generation purposes. This results in the 
synergistic effect that not only are existing peer-to-peer 
tunneling schemes and applications level client server 
security arrangements combined, but they are combined in a 
way which greatly reduces implementation costs 

15 In order to understand the present invention, it is 

necessary to understand a few basic concepts about computer 
to computer communications, including the concepts of 
"layers" and communications protocols, and of mutual 
authentication and file encryption. Further information 

20 about layers and protocols can be found in numerous sources 
available on the Internet, a few of which are listed at the 
end of this section, while a detailed description of a 
mutual authentication and encryption system and method 
suitable for use in connection with the present invention 

25 can be found in U.S. Patent No. 5,602,918, which is 
incorporated herein by reference. In general, the basic 
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communications protocols and architecture used by the 
present invention, as well as authentication , encryption, 
and key management schemes, are already well-known, and can 
be implemented as a matter of routine programming once the 
5 basic nature of the invention is understood. The changes 
made by the present invention to the conventional client 
server virtual private network may be thought of as, 
essentially, the addition of means, most conveniently 
implemented as shims, which add a secured mutual 
10 authentication and session key generation channel between 
the server and all parties to a communication, at all 
levels at which a communication can be carried out. 

Having explained the key differences between the 
present invention and existing systems, the basic concepts 

15 of layers and so forth will now be briefly explained by way 
of background. First, the concept of "layers," "tiers," 
and " levels," which essential to an understanding of the 
invention, simply refers to libraries or sets of software 
routines for carrying out a group of related functions, and 

2 0 which can conveniently be shared or called on by different 
programs at a higher level to facilitate programming, 
avoiding duplication and maximizing computer resources. 
For example, the Windows NT device driver architecture is 
made up of three basic layers, the first of which is the 

25 Network Driver Interface Specification (NDIS 3.0) layer, 
the second of which is called the Transport Driver 
Interface (TDI) layer, and the third being the file 
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systems. These layers are generically referred to as the 
network driver layer, the transport or transport driver 
layer, and the applications layer. 



In the Windows NT architecture, the TDI layer formats 
5 data received from the various file systems or applications 
into packets or datagrams for transmission to a selected 
destination over the open network, while the NDIS layer 
controls the device drivers that send the data, packets f or 
IP datagrams , for example by converting the stream of data 
10 into a waveform suitable for transmission over a telephone 
line or a twisted pair cable of the type known as an 
Ethernet, 

By providing layers in this manner, an applications 
software programmer can design an application program to 

15 supply data to the TDI layer without having to re-program 
any of the specific functions carried out by that layer , 
and all of the transmission, verification, and other 
functions required to send a message will be taken care of 
the TDI layer without further involvement by the 

2 0 applications software. In a sense, each "layer" simply 
accepts data from the higher layer and formats it by adding 
a header or converting the data in a manner which is 
content independent, with retrieval of the data simply 
involving reverse conversion or stripping of the headers , 

25 the receiving software receiving the data as if the 
intervening layers did not exist. 
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In the case of Internet communications , the most 
commonly used set of software routines for the transport or 
TDI layer, which takes care of the data formatting and 
addressing, is the TCP/IP protocol, in which the transport 
control protocol (TCP) packages the data into datagrams and 
provides addressing, acknowledgements, and checksum 
functions, and the internet protocol (IP) further packages 
the TCP datagrams into packets by adding additional headers 
used in routing the packets to a destination address. 
Other transport protocols which can be included in the TDI 
layer include the user diagram protocol (UDP), the internet 
control message protocol (ICMP), and non-IP based protocols 
such as Netbeui or IPX. 

Additional "protocols" are may be used at the 
applications level, although these protocols have nothing 
to do with the present invention except that they may be 
included in the applications programs served by the 
network. Common applications level protocols which utilize 
the TCP/IP protocol include hypertext transfer protocol 
(HTTP), simple mail transfer protocol (SMTP), and file 
transfer protocol (FTP), all of which operate at the layer 
above the transport layer. 

Some applications are written to directly call upon 
the TCP functions. However, for most applications 
utilizing a graphical user interface conveniently rely on 
a set of software routines which are considered to operate 
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above the TDI layer , and are known as sockets. Sockets 
serve as an interface between the TCP set of functions, or 
stack, and various applications, by providing libraries of 
routines which facilitate TCP function calls, so that the 
5 application simply has to refer to the socket library in 
order to carry out the appropriate function calls. For 
Windows applications, a commonly used non-proprietary 
socket is the windows socket, known as Winsock, although 
sockets exist for other operating systems or platforms, and 
10 alternative sockets are also available for Windows, 
including the Winsock 2 socket currently under development. 

In order to implement a virtual private network, the 
encryption and authentication functions must be carried out 
at one of the above "levels," for example by modifying the 

15 network drivers to encrypt the IP datagrams, by inserting 
authentication headers into the TCP/IP stacks, or by 
writing applications to perform these functions using the 
existing drivers. If possible, it is generally desirable 
to minimize modification of the existing levels by adding 

20 a layer to perform the desired functions, calling upon the 
services of the layer below, while utilizing the same 
function calls so that the higher layer also does not need 
to be modified. Such a layer is commonly referred to as a 
"shim. " 

25 As indicated above, the preferred approach to 

implementing client/ server virtual private networks is to 
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use an applications level security system to encrypt files 
to be transmitted, and to then utilize existing 
communications layers such as winsock, or TCP/IP directly. 
This is the approach taken by the commercially available 
access control system known as SmartGATE™, developed by V- 
One Corp. of Germantown, Md., which provides both 
encryption and mutual authentication at the applications 
level utilizing a dedicated server known as an 
authentication server and authentication client software 
installed at the applications level on the client 
computers . A description of the manner in which encryption 
and mutual authentication is carried out may be found in 
the above-cited U.S. Patent No. 5,602,918. While the 
principles of the invention are applicable to other 
client/ server based virtual private networks, SmartGATE™ is 
used as an example because it provides the most complete 
range of mutual authentication and encryption services 
currently available. 

The present invention can be implemented using the 
existing SmartGATE™ system, but adds mutual authentication 
and encryption services to lower layers by intercepting 
function calls or data packets and, during initialization 
of a communications link, establishing separate channels 
between the party initiating the communication and the 
authentication server, and between the authentication 
server and the party which is to share in the 
communication, so as to mutually authenticate the parties 
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with respect to the server , and so as to establish a 
session key which can be used for further direct 
communications between the parties . 

A number of protocols exist which . can be used, in 
5 total or in part, to implement the mutual authentication 
and encryption services at the lower layers, using the same 
basic authentication and encryption scheme currently 
implemented by SmartGATE™ at the applications level. These 
include, by way of example, the SOCKS protocol, which 
10 places a shim between the TDI or transport layer and the 
applications, and the commercially available program, known 
as SnareNet, which operates at the network driver level and 
can be directly utilized in connection with the present 
invention. 



15 On the other hand, a network level implementation such 

as the SKIP protocol, which operates below the TDI layer to 
encrypt the datagrams, and which in its description 
explicitly precludes the generation of session keys (see 
the above cited U.S. Patent No. 5,602,918), is 

2 0 fundamentally different in concept than the present 
invention. Similarly, alternative implementations such as 
Point-to-Point Tunneling Protocol (PPTP) which involve 
modifying the TCP/IP stack and/or hardware to provide 
encryption, as opposed to inserting shims, are not utilized 

25 by the preferred embodiment of the present invention, 
although individual aspects of the protocol could perhaps 
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be used, and the present system could be added to computers 
also configured to accept PPTP communications. 

The SmartGATE™ system uses public key and DES 
encryption to provide two-way authentication and 56-bit 
5 encrypted communications between a server equipped with the 
SmartGATE program and client computers equipped with a 
separate program. Currently, SmartGATE™ operates at the 
highest level, or applications level, by using shared 
secret keys to generate a session key for use in further 

10 communications between the authentication server or gateway 
and the client program. Since the session key depends on 
the secret keys at the gateway and client sides of the 
communication, mutual authentication is established during 
generation of the session key, which can then be used to 

15 encrypt further communications . 

When installed on a client system, the SmartGATE™ 
client software reads a request for communications by an 
applications program, such as a browser program, and then 
proceeds to establish its own communications link with the 

20 destination server to determine if the server is an 
authentication server. If it is not, control of 
communications is relinquished, but if. it is, then the 
security program and the server carry out a 
challenge/response routine in order to generate the session 

25 key, and all further communications are encrypted by the 
security program. Although this program is placed between 
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the Winsock layer and the applications, it does not 
function as a shim, however, because it only affects 
communications directed to the authentication server • 



Having briefly summarized the concepts used by the 
5 present invention, including the concepts of layers , 
protocols, and shims, and having described a specific 
applications level security program which is to be modified 
according to the present invention by adding shims in a way 
:Q which enables secured authentication and session key 

m 10 generation channels to be set up from the lower layers , it 
is] should now be possible to understand the nature of the 

invention, and in particular how it integrates the two 
J;^ approaches to virtual private networking in a way which 

^ greatly expands the concept and yet can easily be 

15 implemented. More details will be given below, but as a 
v3 final observation in this background portion of the patent 

specification, it should be noted that while the overall 
concept of the invention is in a sense very simple, it is 
fundamentally at odds with present approaches. For 
20 example, the literature is replete with references to 
conflicts between VPN standards and implementations , as 
exemplified by the title of an article from LAN Times On- 
Line, 9/96, (http://www.wcmh.com/), which reads Clash Over 
VPN Supremacy. Even a cursory search of the available 
25 literature indicates that the amount of information and 
choices available to those wishing to set up a virtual 
private network is overwhelming. One can choose between 
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Netscape Communications Secure Socket Layer, Open Market 
Inc. 's Secure HTTP , Microsoft's PPTP, among others. 
However, all of these approaches operate at a single level, 
and force a choice between establishing a network of the 
5 type shown in Fig. 1A and a network of the type shown in 
Fig. IB. Only the present invention offer the advantages 
of both approaches, without the inflexibility of 
client/ server arrangements or the costs of more distributed 
architectures . 



10 For further information on the various competing VPN 

protocols and systems, see also The Development of Network 
Security Technologies , Internet Smartsec, 2/97 
(http://www.smartsec.se), which compares SmartGATE™ to 
other application level security systems, including PPTP, 

15 SSL, and S-HTTP; Point-To-Point Tunneling Protocol (PPTP) 
Frequently Asked Questions , Microsoft Corp., date unknown, 
(http://www.microsoft.com), Simple Key-Management for 
Internet Protocols (SKIP), Aziz et al . , date unknown, 
(http://skip.incog.com), and SOCKS Protocol Version 5, RFC 

20 1928, Leech et al . , 3/96 (http://andrew2.andrew.cmu.edu) 
(this document describes a protocol involving a TDI shim). 
For more general information on security problems, Internet 
protocols, and sockets, see Introduction to the Internet 
Protocols, Charles L. Hedrick, Rutgers University, 1987 

25 (http://oac3.hsc.uth.tmc.edu); Windows Sockets - Where 
Necessity is the Mother of Reinvention, Stardust 
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Technologies, Inc., 1996, (http//www. Stardust, com) , and 
Secure Internet Connections , LAN Times r 6/17/96 (Ibid). 



SUMMARY OF THE INVENTION 

It is accordingly a principal objective of the 
5 invention to provide a client/ server virtual private 
network which is capable not only of carrying out 
authenticated secure communications over an open network 
between an authentication server and clients, but also 
authenticated secure peer-to-peer communications. 

10 It is also an objective the invention to provide a 

virtual private network that provides data encryption and 
mutual authentication for both client/server and peer-to- 
peer communications for different-types of applications/ 
using both the applications level and lower levels of a 

15 communications hierarchy. 

It is a further objective of the invention to provide 
a client/ server virtual private network which can provide 
both client/ server and peer-to-peer encryption and 
authentication services for any application sharing a 
20 specified socket or sockets, whether or not the application 
is recognized by the encryption and authentication program. 

It is a still further objective of the invention to 
provide a client/ server virtual private network which can 
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provide encryption and authentication services at the 
applications level, transport driver interface level, and 
network interface level, without the need for modifying 
either the communication driver or network driver, or any 
5 sockets utilizing the communications driver interface. 

It is yet another objective of the invention to 
provide a virtual private network which provides encryption 
and authentication services for peer-to-peer communications 
while maintaining centralized control of key distribution 
10 and management functions. 

Finally, it is also an objective of the invention to 
provide a virtual private network which provides encryption 
and authentication services for peer-to-peer communications 
and in which registration is carried out by a central 
15 gateway server. 

These objectives of the invention are accomplished by 
providing a virtual private network for communicating 
between a server and clients over an open network and in 
which the clients are equipped with an applications level 

20 encryption and mutual authentication program which includes 
at least one shim positioned above either the socket, 
transport driver interface, or network interface layers of 
a client computers communications hierarchy, and which 
intercepts function calls or data packets in order to 

25 authenticate the parties to the communication by 
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establishing secured channels between the server and the 
parties to the communication, prior to establishment of the 
secured communications link between the parties, in order 
to carry out mutual authentication and session key 
5 generation functions. 

More particularly, according to the principles of a 
preferred embodiment of the invention, client 
communications software is provided which, at the socket or 
transport driver interface levels, intercepts function 

10 calls to the socket or transport driver and directs calls 
to the authentication server in order to perform encryption 
and authentication routines, and at the network driver 
interface, performs encryption and authentication functions 
by intercepting the datagrams or data portions of the 

15 packets transmitted by the transport driver interface based 
on communications between the authentication server and the 
client. According to this aspect of the invention, a 
system of providing authentication and encryption services 
for the purpose of establishing a virtual private network 

2 0 includes a plurality of shims arranged to operate at 
different protocol levels in order to establish a common 
secure communications link to an authentication server. 

In one especially preferred embodiment of the 
invention, the client software includes a Winsock shim 
25 arranged to intercept function calls to the Winsock library 
on a client machine and redirect initial communications 
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through the authentication client software to the 
authentication server, so that any function calls to the 
Winsock library of programs are intercepted by the shim and 
carried out by the applications level security program. In 
this embodiment, the client authentication software 
substitutes its own function calls for the original 
function calls in order to establish a secured 
communications link to the authentication server over which 
such functions as mutual authentication between the client 
and server, indirect authentication of peer applications by 
the now trusted server, session key generation, are carried 
out, as well as ancillary functions such as on-line 
registration (OLR), utilizing the unmodified original 
Winsock library and TCP/IP communications stacks. 



By inserting a shim at the Winsock level, an 
applications level client/server based security program 
such as SmartGATE™ can be used to provide secure 
communications for any application which utilizes the 
Winsock library. In addition, by including analogous shims 
at other levels, the invention can be used to secure 
virtually any communications application, including those 
which by-pass the TDI layer and communicate directly with 
the network driver level. 



Instead of the current array of mutually exclusive 
alternative methods and systems of establishing secured 
communications over an open network, the invention thus 
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provides a single integrated method and system capable of 
carrying out both client/ server communications and peer-to- 
peer communications between a wide variety of 
communications applications regardless of whether the 
applications use a socket or even commonly accepted 
internet protocols, with complete mutual authentication and 
encryption of data files at all levels and between all 
parties to the network. 



It will be appreciated that the term "virtual private 
network" is not to be taken as limiting , and that the 
principles of the invention can be applied to any remote 
access schemes which utilize the Internet or other 
relatively insecure networks to provide access for remote 
users, corporate intranets, and electronic commerce. 



BRIEF DESCRIPTION OF THE DRAWINGS 



Fig. 1A is a schematic diagram of a client/server 
virtual private network. 




Fig. VB is a schematic diagram of an alternative 
virtual private network based on peer-to-peer 



communications . 

/ 




Fig. 2 is a functional block diagram showing the 
operation of an applications level security program in a 
conventional communications network hierarchy. 
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Fig. 




is a functional block diagram showing the 



communications network hierarchy of Fig. 1, modified to 
provide a second layer of service in accordance with the 
principles of a preferred embodiment of the invention. 



Fig. 4 is a functional block diagram showing the 
communications network hierarchy of Fig. 2, modified to 
provide a third layer of service in accordance with the 
principles of the preferred embodiment. 



communication network hierarchy of Fig. 3, modified to 
provide a fourth layer of service in accordance with, the 
principles of the preferred embodiment. 



Fig. 6 is a schematic diagram of a virtual private 
network utilizing the principles of the preferred 
embodiment of jfhe^ invention. 



Fig. 7 is a flowchart illustrating a method of 
implementing the system of the preferred embodiment. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

Fig. 2 illustrates the operation of a client 
authentication program which is utilized in the present 
invention. An example of such a program is the SmartGATE™ 
program discussed briefly above , although other 





a functional block diagram showing the 
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applications level security programs , whether or not token 
based , could be modified in a manner similar to that 
discussed in the following description. The illustrated 
hierarchy is the Windows NT architecture, although versions 
5 of SmartGATE™ exist for other architectures, and the 
invention could easily be adapted for use with any version 
of SmartGATE™, including UNIX and Macintosh versions, as 
well as for use with applications level security programs 
designed for communications architectures other than those 

10 supported by SmartGATE™. Conversely, it is intended that 
the present invention can be used with authentication and 
encryption schemes other than that used by SmartGATE™ and 
disclosed in U.S. Patent No- 5,602,918. For purposes of 
convenience, therefore, the software represented by 

15 SmartGATE™ is simply referred to as client authentication 
software . 

In addition, it noted that the client computer 
architectures illustrated in Figs- 3-6, which are modified 
versions of the architecture of Fig. 2, is to be used with 

20 an overall network layout such as the one illustrated in 
Fig. 6, which includes an authentication server that may be 
a SmartGATE™ server, or another server depending on the 
client authentication software. The invention is not 
merely the addition of shims to the client software, but 

25 involves the manner in which the shims are used in the 
establishment of the authentications and key generation 
links to the server. 



20 




Turning to Fig. 2, which provides background for the 
description of the invention illustrated in Figs. 3-6 f the 
client authentication software 2 0 is situated above the 
boundary of the transport or TDI layer 21 and is designed 
5 to utilize a socket 22, such as Winsock, to carry out 
communications with the authentication server 23 shown in 
Fig. 6 by means of a transport protocol such as TCP/IP, 
UDP, or the like, which in turn supply datagrams or packets 
to a hardware driver layer 24, such as NDIS 3.0, of a 
10 network or modem connection 25. 

In operation, the client authentication software 20 
intercepts interconnect calls 2 6 form client authentication 
software supported applications 2 7 and, if the calls are 
directed to the authentication server 23, or to a server 28 

15 situated on a secured network whose access is controlled by 
the authentication server, establishes a secured 
communications link to the server by executing appropriate 
function calls 2 9 to the socket library, which in turn 
transmits function calls 30 to the TDI layer, causing the 

20 TDI layer to form datagrams or packets 31. Datagrams or 
packets 31 are then formatted over packaged for 
transmission by the hardware drivers 24 and sent to the 
communications network in the form of Ethernet packets or 
analog signals 32 containing the original datagrams from 

25 the TDI layer. Once the secured communications link has 
been established, client authentication software 20 
encrypts all further data communications 34 from 
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applications 27, which are indicated by dashed lines f 
before handing them off to the next lower layer in the form 
of encrypted files 35. The dashed lines are shown in Fig. 
2 as extending only to the TDI layer 21 , because the 
5 datagrams formed by the TDI layer are indistinguishable as 
to content, but it is to be understood that datagrams or 
packets 31 carry both the communications used to establish 
the secure channel, and the encrypted files subsequently 
sent therethrough . 

10 Finally, in the case of SmartGATE™, the 

authentication client software utilizes either a smart card 
or secured file to supply the secret keys used during 
authentication to generate a session key for encryption of 
further communications, and also to carry out certain other 

15 encryption and authentication functions, although it is of 
course within the scope of the invention to use key 
distribution and authentication methods which do not rely 
on smartcards or tokens, and the tokens are not involved in 
any of the basic communications functions of the client 

2 0 authentication software 20. 

In addition to the applications 27 which communicate 
with the server via the authentication/encryption software 
20, a typical system will have a number of additional 
software applications 3 6 and 37 capable of carrying out 
25 communications over the open network, but which the 
authentication client software is not configured to handle, 



22 




and which are not specifically adapted or intended to carry 
out communications with the authentication server. These 
are referred to herein as peer-to-peer applications f and 
can include applications which use the same sockets as the 
5 authentication client software, applications which directly 

call upon a transport driver interface stack, whether using 
the same protocol as the authentication client software or 
another protocol, all of which are intended to be 
represented by the TDI layer, and applications which are 

10 written to call directly upon the hardware drivers* These 
peer-to-peer applications may have their own encryption and 
authentication capabilities, but cannot utilize the 
services of the authentication server or client software, 
and therefore the function calls made by the applications 

15 and the files transmitted are indicated by separate 
reference numerals 40-43. 

It will be appreciated by those skilled in the art 
that lower layer application programs which generate 
packets in forms other than those represented by the TDI 

20 layer are also possible, and should be considered within 
the scope of the invention, but at present virtually all 
open network applications use at least one of the TDI 
protocols, and thus while these programs may interact 
directly with the network driver layer, and require a 

25 network driver layer shim, as will be discussed below, are 
illustrated for purposes of convenience as part of the TDI 
layer applications . 
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Turning now to a preferred embodiment of the 
invention, the arrangement shown in Fig. 3 modifies the 
arrangement of Fig. 2 by adding a socket shim 5 0 between 
the socket 22 utilized by the authentication client 
5 software 20, the peer-to-peer applications 3 6 which also 
utilize the socket 20, and the authentication client 
software itself. The shim 5 0 operates by hooking or 
intercepting call initiation function calls 40 made to the 
socket and, in response thereto , having the authentication 

10 client software initiate communications with the 
authentication server 23, shown in Fig. 6, in order to 
carry out the authentication protocol, as will be discussed 
in more detail below. Shim 50 also causes files 41 
intended for the TDI layer to be diverted to the 

15 authentication software for encryption based on the session 
keys generated during the initial communications with the 
authentication server, and transmission as encrypted files 
51 addressed to the peer application, also shown in Fig. 6, 
which could also be an application on the application 

20 server 28. 

Since the basic authentication client software is 
designed to send all communications directly to the 
authentication server, while the peer-to-peer applications 
are designed only to communicate with "peers" 45 and not 
25 with the authentication server, the principal function of 
shim 50 is to arrange for the destination of address of the 
communication to be supplied to both the authentication 



24 




client software and to authentication server , even though 
the peer application assumes that it is communicating only 
with the peer application. This function permits session 
key encrypted communications to be forwarded directly to 
5 the peer application, as illustrated in Fig. 6, while the 
latter function provides the authentication server with the 
client address so that the authentication server can 
establish a secured and authenticated link with the peer 
application , via authentication client software on the peer 
10 computer, and transmit the session key to the peer 
application or at least enable the peer application to 
recreate the session so that it can decrypt the encrypted 
files received directly from the client application. 

Thus, while it is appreciated that the use of socket 
15 shims is well-known, as mentioned above, the socket shim 
shown in Fig. 2 has the unique function of enabling direct 
peer-to-peer communications with mediation by the 
authentication server, permitting the highest level of 
authentication service and collateral functions. In 
2 0 addition, because of the mediation by the key server , the 
peer applications do not need to have a shared secret key, 
allowing centralized key management , with only the 
authentication server having access to all of the client's 
secret keys. 

25 Figs. 4 shows the variation of the client 

authentication software 2 0 in which a TDI shim 52 similar 
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in function to the socket shim 5 0 is provided above the TDI 
layer. Like the socket shim, implementation of the TDI 
shim essentially simply involves diverting certain 
information to the client software in order to establish a 
5 communications link with the authentication server, and 
subsequently perform encryption to obtain encrypted files 
54 for transmission directly through the TDI layer in the 
usual manner. As with the socket shim, TDI shims are not 
new and can be implemented in known manner, by intercepting 
10 TDI service requests, but with the difference from prior 
TDI shims that the TDI shim works with the authentication 
software 2 0 and authentication server to authenticate 
communications and generate a session key. 

Finally, as shown in Fig. 5, a further layer of 
15 authentication and encryption may be added by adding a 
network driver shim 55, either to the arrangement shown in 
Fig. 3 without the TDI shim, in combination with the TDI 
shim shown in Fig. 4, or in combination with the TDI shim 
of Fig. 4 but not the socket shim, to provide for 
20 authentication of communications at the network driver 
layer. At this layer, the shim 55 intercepts IP packets 
from applications 56, but instead of referring back to the 
applications level routine, checks the destination address 
(which can be in TCP format, UDP format, and so forth)/ 
25 establishes a session key by communications with the 
authentication server, converts the session key into a 
format which can be used to encrypt the IP packet, and 




sends the IP packet towards the destination, all by 
carrying out the necessary operations at the network driver 
level, in a manner similar to that utilized by the above- 
mentioned SnareNet software program, but with the 
5 difference that the authenticating communications link and 
key generation is carried out by packets addressed to a 
corresponding layer 5 6 of the authentication server, which 
may be further connected to an applications server 57. 

It will be noted that since the IP packets are not 
10 distinguishable by content, the network driver layer shim 
could be used as an additional level of security, rather 
than as an alternative to applications level encryption, 
with the encrypted files generated by software 20 being 
further encrypted by shim 55 before transmission to the 
15 authentication server or associated gateway. 

The overall system utilizing the authentication client 
software illustrated in Figs. 3-5 is schematically 
illustrated in Fig. 6. The principal components of the 
overall system are the client computers containing software 

20 of the type illustrated in Figs. 2-5, including client 
authentication software 2 0 and shims 50, 53, and/or 55, and 
applications with communications capabilities (represented 
by applications 27, 36, 37, and 56 on one client, and 
application 45 on the other). For purposes of 

25 illustration, the client of Figs. 6 is thus depicted as 
including applications for communicating at the highest 




levels , such as the SmartGATE™ proxy application 
applications for communicating at the network driver level 
with corresponding applications connected to the lower 
layer of the authentication server, and peer-to-peer 
5 applications with no capability of communicating with 
SmartGATE™, but which use sockets or TDI protocols 
recognized by the shims. 

In the case of the SmartGATE™ proxy application f 
communications are established in the same manner as in the 

10 currently available version of the SmartGATE™ 
authentication client software, and as described in U.S. 
Patent No. 5,602,918, the communications link being 
indicated by arrows 6 0 and 61, with arrow 60 representing 
the client/ server response channel used to authenticate the 

15 parties and generate the session key. 

In the case of a peer-to-peer application , in which 
the clients wish to communicate over a direct link 62 , the 
invention provides for the function calls establishing the 
communications to be intercepted and the initialization 

2 0 procedure routed through channel 61 to the authentication 
server 23. Server 23 then opens a secured channel 63 to 
the authentication client software 2 0 associated with peer 
application 45 by performing the same mutual authentication 
procedure performed for the purpose of establishing channel 

25 63 , and once the channel is established with its own 
session key, transmits information using the channel 63 
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session key which allows the client to recreate the channel 
60 session key for use in decrypting communications sent 
over channel 62. Alternatively, after establishing channel 
63, the channel 60 session key could be used to transmit 
back to the original sending party information necessary to 
recreate the channel 63 session key. In either case, the 
authentication server is thus used to establish a fully 
authenticated "tunnel" between the peer applications 
without the need to modify any of the sockets, TDI 
protocols, or hardware drivers on either of the client 
computers, while the transmitting peer application has no 
way of directly authenticating the receiving peer, only a 
receiving peer authenticated by the authentication server 
will be able to generate the necessary session keys, and 
thus each of the parties to the communication is 
effectively authenticated. 

For the lower layer application 56, a similar protocol 
may be employed, in which the attempted communication 
between lower layer applications is intercepted, and the 
communications link to the authentication server is used to 
generate a session key, which is then used to encrypt the 
packets or datagrams being sent. In this case, the 
destination must be the lower layer of the authentication 
server, and thus the communications link is indicated by a 
separate channel 67, 
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Finally, the procedures associated with the network 
illustrated in Fig. 6 are summarized in the flowchart of 
Fig. 7. For communications directly with the applications 
level portion of the server 23 , steps 100-103 are used, 
while for peer-to-peer communications, steps 104-109 are 
used, and for network driver level communications, steps 
110-114 are used. 

In particular, step 100 by which the applications 
level authentication program 2 0 illustrated in Figs. 3-5 
receives a call initiation request, either directly from a 
supported applications program 2 7 or from a programs 36 and 
37 via one of the shims 50 and 53, step 101 is step by 
which the program 2 0 addresses the authentication server, 
step 102 is the step by which the client and server are 
mutually authenticated and the session keys generated 
using, for example, the procedure described in U.S. Patent 
No. 5,602,918, and step 103 is the step by which program 20 
encrypts further communications received directly or via 
shims 5 0 and 5 3 from the applications programs 27, 36, and 
37. 

For peer-to-peer communications, step 105, which is 
part of step 100, is the step by which the peer address is 
supplied to program 20, steps 106 and 107 are identical to 
steps 101 and 102, step 108 is the step by which 
communications channel 63 shown in Figure 6 is established, 
step 109 is the step by which the destination computer 
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authenticated by the server is enabled to decrypt 
communications received over channel 62 , and step 110 is 
the step by which program 2 0 encrypts the communications. 
It will of course be appreciated that these steps represent 
only a summary of the steps involved in carrying out the 
present invention, and that further steps will be apparent 
to those skilled in the art based on the above description 
of the apparatus and software portions of the preferred 
embodiment of the invention. 

Having thus described various preferred embodiments of 
the invention, those skilled in the art will appreciate 
that variations and modifications of the preferred 
embodiment may be made without departing from the scope of 
the invention. It is accordingly intended that the 
invention not be limited by the above description or 
accompanying drawings, but that it be defined solely in 
accordance with the appended claims . 
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